Papers with in-depth evaluation
Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in generative language modeling applied to discrete speech tokens presented a new avenue for text-to-speech (TTS) synthesis. |
| Approach: | They propose to use generative language modeling to generate text-to-speech (TTS) outputs by a discrete token-based model. |
| Outcome: | The proposed model is rated higher in naturalness and context appropriateness in listening tests compared to a conventional TTS. |
Examining the Limitations of Computational Rumor Detection Models Trained on Static Datasets (2024.lrec-main)
Copied to clipboard
| Challenge: | Past research has indicated that content-based rumor detection models perform less effectively on unseen rumors. |
| Approach: | They propose to use data split strategies to minimize the effects of temporal concept drift in static datasets during the training of rumor detection methods. |
| Outcome: | The proposed model over-relys on the information derived from the rumors’ source post and overlooks the significant role that contextual information can play. |
On Classifying whether Two Texts are on the Same Side of an Argument (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to same side stance classification (S3C) require domain knowledge and semantic inference to solve the task. |
| Approach: | They propose to use same side stance classification to predict whether two arguments argue for the same stance for a given pair of arguments. |
| Outcome: | The proposed model fails to generalize both within and across topics and domains when adjusting the sampling strategy to a more adversarial scenario. |
MIBench: Evaluating Multimodal Large Language Models over Multiple Images (2024.emnlp-main)
Copied to clipboard
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu
| Challenge: | Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored. |
| Approach: | They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios. |
| Outcome: | The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC. |
FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Spatial reasoning is a fundamental aspect of human intelligence. |
| Approach: | They propose a framework to assess FoR comprehension in large language models (LLMs) by using the Frame of Reference Evaluation in Spatial Reasoning Tasks benchmark. |
| Outcome: | The proposed method improves overall performance across spatial reasoning tasks. |